{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# PP-SHITU介绍\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/dbea9d82db6449a6a42a6ef3af088716430105035ed847d195bfec8998447468)\n",
    "\n",
    "\n",
    "**超轻量的主体检测**\n",
    "主体检测作为整个识别任务的第一步，其本身的精度、性能， 都直接影响整个识别系统的识别效果。PP-ShiTu中使用PP-PicoDet模型作为主体检测算法，PP-PicoDet模型性能和速度均达到业内SOTA的水平，为整个识别系统实现精准高效识别打下了坚实的基础。\n",
    "\n",
    "**高效的特征提取模块**\n",
    "图像识别的又一大问题就是如何让模型提取到更好的特征。在特征提取的训练阶段，PP-ShiTu通过使用度量学习，更好地解决高相似度物体的区分问题。不仅如此，PP-ShiTu所使用的骨干网络PP-LCNet作为业内SOTA模型，大幅度提升预测速度的同时，还提高了精度，并且可能直接支持多种应用方向和场景，真节省开发成本的一把好手!\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/ccef6ac1913b434682614542a697c838597fbc0fc77d4a03a1efd42cb4c0e351)\n",
    "\n",
    "\n",
    "**快速向量检索支持**\n",
    "在实际应用中，海量的图像、视频特征不仅会消耗巨大的存储空间，而且检索时间极长，给图像识别的最后一公里设下路障。PP-ShiTu则是结合DeepHash和度量学习，甚至在检索库特征数量大于10万时，依然使得所需的存储空间减少32倍，检索速度提高5倍以上。除此以外PP-ShiTu使用的向量搜索模块Faiss，可以更好地适应多平台的需求(Linux, Windows, MacOs)，为实际应用提供灵活选择。\n",
    "这样一个高效系统使用起来却只需三步，绝对的”开箱即用”！\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/d0fc7fa0f3664da7b0d878aecbefcc54be2c65f1ba2649ac97b9e5c392e01f0e)\n",
    "\n",
    "\n",
    "* 第一步通过目标检测模型，进行主体检测；\n",
    "* 第二步对每个候选区域进行特征提取；\n",
    "* 第三步将特征提取后的向量在检索库中进行检索，完成匹配，返回识别结果。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 下载模型和DEMO数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 下载模型以及demo数据, 参考文档: https://github.com/PaddlePaddle/PaddleClas/blob/release/2.3/docs/zh_CN/quick_start/quick_start_recognition.md\n",
    "!mkdir models\n",
    "%cd models\n",
    "\n",
    "# 下载通用检测 inference 模型并解压\n",
    "!wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar && tar -xf picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar\n",
    "\n",
    "# 下载识别 inference 模型并解压\n",
    "!wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/general_PPLCNet_x2_5_lite_v1.0_infer.tar && tar -xf general_PPLCNet_x2_5_lite_v1.0_infer.tar\n",
    "\n",
    "%cd ../\n",
    "# 下载 demo 数据并解压\n",
    "!wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/data/drink_dataset_v1.0.tar && tar -xf drink_dataset_v1.0.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 基于DEMO数据构造Gallery索引\n",
    "\n",
    "下载好的demo数据已经包含了索引文件，若需重新构建索引可以运行命令:\n",
    "\n",
    "```\n",
    "下面是使用下载的服务端商品识别模型进行索引库构建\n",
    "python3.7 python/build_gallery.py -c configs/inference_general.yaml -o Global.rec_inference_model_dir=./models/general_PPLCNet_x2_5_lite_v1.0_infer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 测试图片效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/pp-shitu_demo\n",
      "[{'bbox': [409, 109, 668, 968], 'rec_docs': '农夫山泉-饮用天然水', 'rec_scores': 0.51558024}]\n"
     ]
    }
   ],
   "source": [
    "%cd /home/aistudio/pp-shitu_demo\n",
    "!python main.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "图片路径: pp-shitu_demo/results.jpg\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/388bb070d76e4023b00c73a366d6fb701abeedcad7bb49c79765cfb3f52e37f8)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 应用场景举例： 跨镜头搜索\n",
    "\n",
    "某大牛侦探发现一名逃犯驾车驶离犯罪现场，此时侦探已掌握一个犯罪现场视频片段，需要快速在各个摄像头下搜寻到逃离车辆。侦察科技术员飞飞采取的技术方式如下：\n",
    "\n",
    "1. 通过目标跟踪，提取出目标车辆的特征\n",
    "2. 获取各个摄像头的单帧画面，将所有车辆与目标车辆进行特征匹配\n",
    "3. 找到最佳特征匹配的候选车辆， 进一步再人工筛选或者车牌OCR辅助识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 基于PP-tracking 跨镜跟踪快速构建图片集\n",
    "\n",
    "PP-Tracking是基于飞桨深度学习框架的业界首个开源实时跟踪系统。针对实际业务的难点痛点，PP-Tracking内置行人车辆跟踪、跨镜头跟踪、多类别跟踪、小目标跟踪及流量计数等能力与产业应用，同时提供可视化开发界面。模型集成多目标跟踪，目标检测，ReID轻量级算法，进一步提升PP-Tracking在服务器端部署性能。同时支持python，C++部署，适配Linux，Nvidia Jetson多平台环境。\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/c8ffcde8c1974a69863806e3aeaf81c6680f3a8de50d4d56be9a14f155979f9e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#下载并安装依赖\r\n",
    "# !git clone https://gitee.com/paddlepaddle/PaddleDetection.git -b develop\r\n",
    "# !pip3 install --upgrade pip -i https://mirror.baidu.com/pypi/simple\r\n",
    "%cd /home/aistudio/PaddleDetection/ \r\n",
    "!python -m pip install pycocotools -i https://mirror.baidu.com/pypi/simple && python -m pip install -r requirements.txt -i https://mirror.baidu.com/pypi/simple && python setup.py install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 下载预测部署模型(车辆检测PicoDet)以及PPLCNet车辆ReID模型\r\n",
    "%cd /home/aistudio/PaddleDetection\r\n",
    "!wget https://paddledet.bj.bcebos.com/models/mot/deepsort/picodet_l_640_aic21mtmct_vehicle.tar\r\n",
    "!tar -xvf picodet_l_640_aic21mtmct_vehicle.tar\r\n",
    "!wget https://paddledet.bj.bcebos.com/models/mot/deepsort/deepsort_pplcnet_vehicle.tar\r\n",
    "!tar -xvf deepsort_pplcnet_vehicle.tar\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 下载测试视频\r\n",
    "%cd  /home/aistudio/PaddleDetection\r\n",
    "!wget https://bj.bcebos.com/v1/paddledet/data/mot/demo/mtmct-demo.tar && tar xvf mtmct-demo.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 推理测试视频\r\n",
    "%cd /home/aistudio/PaddleDetection\r\n",
    "# 修改deploy/pptracking/python/mot_sde_infer.py, 保存相应的裁剪图片以及特征到文件夹, output/target_images, output/target_features\r\n",
    "!cp /home/aistudio/resources/custom_script.py /home/aistudio/PaddleDetection/deploy/pptracking/python/custom_script.py\r\n",
    "!python deploy/pptracking/python/custom_script.py --model_dir=picodet_l_640_aic21mtmct_vehicle/ --reid_model_dir=deepsort_pplcnet_vehicle/ --device=GPU --scaled=True --threshold=0.5 --video_file=mtmct-demo/c004.mp4 --save_mot_txts --save_images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "图片路径: resources/00095.jpg\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/18d105f6629a47299f122404459b2f4b5a327998085142279e427d66c084166c)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 假设跟踪ID为21的红色车辆是我们要找的目标\n",
    "\n",
    "保存跟踪ID=21的所有特征到numpy文件, 作为数据的Gallery索引\n",
    "\n",
    "统计裁剪后图片的宽高尺寸，可视化查看, 过滤较小的图片宽高(阈值是60像素)\n",
    "\n",
    "选取对向摄像头C003.mp4的一帧图片, 然后在当前参考图片上搜索关注的目标\n",
    "\n",
    "因特征数量有限，这里可以采用贪婪匹配的方式就可以计算最符合特征的目标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28 28\n"
     ]
    }
   ],
   "source": [
    "# 选取红色车辆的图片构建特征\r\n",
    "import cv2\r\n",
    "import os\r\n",
    "import numpy as np\r\n",
    "target_path = \"/home/aistudio/PaddleDetection/output_features/21/\"\r\n",
    "files = os.listdir(target_path)\r\n",
    "\r\n",
    "images = []\r\n",
    "features = []\r\n",
    "for f in files:\r\n",
    "    if f.endswith(\".jpg\"):\r\n",
    "        img_path = os.path.join(target_path, f)\r\n",
    "        im = cv2.imread(img_path)\r\n",
    "        if im.shape[0] < 70 or im.shape[1] <70:\r\n",
    "            continue\r\n",
    "\r\n",
    "        im2 = cv2.resize(im, (100, 100))\r\n",
    "        images.append(im2)\r\n",
    "\r\n",
    "        npy_path =img_path.replace(\".jpg\", \".npy\")\r\n",
    "        features.append(np.load(npy_path))\r\n",
    "\r\n",
    "print(len(images), len(features))\r\n",
    "\r\n",
    "img0 = np.concatenate(images[:3],1)           #沿1轴横向拼接\r\n",
    "img1 = np.concatenate(images[3:6],1)\r\n",
    "img2 = np.concatenate(images[6:9],1)\r\n",
    "img9 = np.concatenate([img0,img1,img2],0)   #沿0轴纵向拼接\r\n",
    "cv2.imwrite(\"/home/aistudio/resources/targets.jpg\",img9)\r\n",
    "np.save(\"/home/aistudio/resources/targets\", np.asarray(features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "图片路径 resources/targets.jpg\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/078ab419100647b399f23d44a903110cae0b4a7019c14158924d50e3d1bc75be)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# !ffmpeg -i /home/aistudio/PaddleDetection/mtmct-demo/c003.mp4 -r 30  -vcodec copy -an -f image2  /home/aistudio/tmp/%05d.jpg\n",
    "# 选取并保存一张参考图片\n",
    "import cv2\n",
    "import random\n",
    "cap = cv2.VideoCapture(\"/home/aistudio/PaddleDetection/mtmct-demo/c003.mp4\")\n",
    "frames_num=cap.get(7)\n",
    "num =110\n",
    "print(\"num\", num, frames_num)\n",
    "\n",
    "cnt = 0\n",
    "while cap.isOpened():\n",
    "    ret,frame = cap.read()\n",
    "    if cnt == num:\n",
    "        cv2.imwrite(\"/home/aistudio/resources/ref.jpg\", frame)\n",
    "        break\n",
    "    cnt += 1\n",
    "    \n",
    "cap.release()\n",
    "cv2.destroyAllWindows()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "图片路径 resources/ref.jpg\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/bafb49ea101e4226ac8b4a70cd0178cefa9959afae6742fdada87718446567f6)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "特征匹配最大值:  0 0.8121714707641735\n",
      "坐标信息: (15, 4) [1371.7288   354.4732   205.90564  140.6035 ]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 特征贪婪搜索\r\n",
    "import numpy as np\r\n",
    "targets_npys = np.load(\"/home/aistudio/resources/targets.npy\")\r\n",
    "ref_npys = np.load(\"/home/aistudio/resources/ref_feats.npy\")\r\n",
    "def cos_dist(x, y):\r\n",
    "    res = np.array([[x[i] * y[i], x[i] * x[i], y[i] * y[i]] for i in range(len(x))])\r\n",
    "    return sum(res[:, 0]) / (np.sqrt(sum(res[:, 1])) * np.sqrt(sum(res[:, 2])))\r\n",
    "\r\n",
    "max_vals = []\r\n",
    "for r in range(ref_npys.shape[0]):\r\n",
    "    max_val = -99\r\n",
    "    for t in range(targets_npys.shape[0]):\r\n",
    "        sim = cos_dist(targets_npys[t], ref_npys[r].T)\r\n",
    "        if max_val < sim:\r\n",
    "            max_val = sim\r\n",
    "    max_vals.append(max_val)\r\n",
    "\r\n",
    "max_index = max_vals.index(max(max_vals, key = abs))\r\n",
    "print(\"特征匹配最大值: \", max_index, max_vals[max_index])\r\n",
    "\r\n",
    "tlwh_npy = np.load(\"/home/aistudio/resources/ref_tlwh.npy\")\r\n",
    "print(\"坐标信息:\", tlwh_npy.shape, tlwh_npy[max_index])\r\n",
    "\r\n",
    "img = cv2.imread(\"/home/aistudio/resources/ref.jpg\")\r\n",
    "x1, y1, w, h = tlwh_npy[max_index]\r\n",
    "intbox = tuple(map(int, (x1, y1, x1 + w, y1 + h)))\r\n",
    "cv2.rectangle(img, intbox[0:2], intbox[2:4], color=(0, 0, 255), thickness=3)\r\n",
    "cv2.imwrite(\"/home/aistudio/resources/result_ref.jpg\", img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 可视化搜索到的跨镜头目标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "图片路径 resources/result_ref.jpg\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/fa52691fcf0f40ba9bc9d966d8438c9d13ffe037a6484cd189c73cd23acc7a67)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "请点击[此处](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576)查看本环境基本用法.  <br>\n",
    "Please click [here ](https://ai.baidu.com/docs#/AIStudio_Project_Notebook/a38e5576) for more detailed instructions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
